Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

_TAGS for fine-grained tasks #703

Open
wants to merge 20 commits into
base: main
Choose a base branch
from
Open

_TAGS for fine-grained tasks #703

wants to merge 20 commits into from

Conversation

sg-wbi
Copy link
Collaborator

@sg-wbi sg-wbi commented Jun 8, 2022 •
edited
Loading

INTRO

This introduces a metadata attribute _TAGS.

The values in _TAGS are, well tags, to further classify the task.

These tags are meant to be used together w/ _SUPPORTED_TASKS.

I tried to be as generic as possible in order to have a as minimal as possible set of tags, whose combination w/ _SUPPORTED_TASKS could make sense.

For instance, bc5cdr will have the following combination of fine-grained tasks:

  • NER Chemical
  • NER Disease
  • NED Chemical
  • NED Disease
  • Relation Disease
  • Relation Chemical

This is just a temporary solution and we can discuss which tags make sense and which do not.

HOW:

You can access these tags by checking out this PR:

gh pr checkout https://github.com/bigscience-workshop/biomedical/pull/703

As mentioned these tags are meant to be composed w/ _SUPPORTED_TASKS:

configs = BigBioConifgHelpers()

for config in configs:
   for task in config.tasks:
       for tag in config._py_module._TAGS:
           finegrined_task = f"{task} {tag}"

TROUBLESHOOTING:

The BigBioConfigHelper should load just fine. If this is not the case, it is probably because of a spelling error.

To fix this you just need to edit the file you find in

bigbio/utils/resources/tags.json

If you find such errors, please ping me on slack and I will fix them right away.

@sg-wbi
Copy link
Collaborator Author

sg-wbi commented Jun 8, 2022

The idea moving forward would be to attach specific tags to specific taks, this way we can have a test for this information, e.g. the "MULTIPLE_CHOICE" tag should be available only for "QA".

@sg-wbi
Copy link
Collaborator Author

sg-wbi commented Jun 8, 2022

One more thing: during the process I was tempted to create a SOCIAL_MEDIA and CLINICAL tag, but I think we should have yet another metadata attribute specific only for "domain"/"source".

@hakunanatasha hakunanatasha self-assigned this Jul 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hakunanatasha hakunanatasha Awaiting requested review from hakunanatasha hakunanatasha is a code owner

@jason-fries jason-fries Awaiting requested review from jason-fries jason-fries is a code owner

@sunnnymskang sunnnymskang Awaiting requested review from sunnnymskang sunnnymskang is a code owner

@ruisi-su ruisi-su Awaiting requested review from ruisi-su ruisi-su is a code owner

@galtay galtay Awaiting requested review from galtay galtay is a code owner

@leonweber leonweber Awaiting requested review from leonweber leonweber is a code owner

@debajyotidatta debajyotidatta Awaiting requested review from debajyotidatta debajyotidatta is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@hakunanatasha hakunanatasha

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sg-wbi @hakunanatasha